Locality-aware parallel block-sparse matrix-matrix multiplication using the Chunks and Tasks programming model

We present a library for parallel block-sparse matrix-matrix multiplication on distributed memory clusters. By using a quadtree matrix representation data locality is exploited without any prior information about the matrix sparsity pattern. A distributed quadtree matrix representation is straightforward to implement due to our recent development of the Chunks and Tasks programming model [Parallel Comput. 40, 328 (2014)]. The quadtree representation combined with the Chunks and Tasks model leads to favorable weak and strong scaling of the communication cost with the number of processes, as shown both theoretically and in numerical experiments. Matrices are represented by sparse quadtrees of chunk objects. The leaves in the hierarchy are block-sparse submatrices. Sparsity is dynamically detected by the matrix library and may occur at any level in the hierarchy and/or within the submatrix leaves. In case Graphics Processing Units (GPUs) are available, both CPUs and GPUs are used for leaf-level multiplication work, thus making use of the full computing capacity of each node. The performance of our matrix-matrix multiplication implementation is evaluated for dense and banded matrices and matrices with randomly distributed submatrix blocks, as well as matrices from linear scaling electronic structure calculations. We show that, compared to methods that do not exploit data locality, our locality-aware approach reduces communication significantly, achieving essentially constant communication per node in weak scaling tests.